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Executive summary 


Recent years have seen a growing recognition 
that social and economic factors play a crucial 
role in influencing academic success and that 
young people require supports that extend 
beyond the school walls. The notion of a 
"learning ecosystem" is taking root, a concept 
that places students at the center, with resources 
and supports organized around them. In such a 
system, schools are extremely important but are 
just one of many influences in a young person's 
life. In addition to schools, the ecosystem 
includes the influence of families and peers; 
out-of-school-time offerings such as afterschool 
programs; and community resources such as 
science centers, libraries and media. 

Afterschool programs have long aimed to 
positively influence students' personal 
development and support their social and 
emotional growth. Studies of high-quality 
afterschool programs have found that 
participating children see a significant 
improvement in their self-perception, increased 
positive social behavior and a decrease in 
problem behaviors— changes that ultimately 
extend to school-related behavior. Hence, 
afterschool programs are increasingly seen as a 
key social and academic support for youth. 

Afterschool researchers and providers have 
recognized that hands-on, inquiry-driven science, 
technology, engineering and math (STEM) 
programs are in line with afterschool's overall 
approach to education and have increasingly 
begun to include STEM in their afterschool 
offerings. Afterschool programs that provide 


strong STEM learning experiences are making an 
impact on participating youth— youth not only 
become excited and engaged in these fields but 
develop STEM skills and proficiencies, come to 
value these fields and their contributions to 
society, and— significantly— begin to see 
themselves as potential contributors to the STEM 
enterprise. 

For example: 

• Program attendance and engagement in the 
4-H Tech Wizards program is high— on 
average 95 percent of enrolled youth attend 
sessions regularly and 95 percent of 
participants stayed in the program for three 
years to build successively on their tech skills. 

• Girlstart After School participants engage 
weekly in rigorous problem-solving. Using 
inquiry and iterative design, they generate 
predictions and hypotheses, document 
observations, and prototype solutions. At the 
end of the year, 91 percent of participants 
demonstrated mastery of scientific inquiry 
and the engineering design process. 

• Every week at Science Club, an afterschool 

program run as a partnership between 
Northwestern University and the Boys & Girls 
Club of Chicago, youth work alongside 
scientist mentors on fun, challenge-based 
curricula. These units focus on health and 
biomedical careers and development of 
scientific skills: experimental design, 

variables, data analysis and evidence-based 
conclusions. As a result, 100 percent of 
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Science Club youth see science as important 
for their future careers, 30 percentage points 
higher than control youth. Two independent, 
well-controlled assessment methods reveal 
increases in participants' scientific skills 
ranging from 25-30 percent over 
non-participating students. 

Participants in Project GUTS, an afterschool 
computing program, come to understand that 
STEM is integral to everyday life and can be 
used to study and potentially solve local 
community problems. Project GUTS 
curriculum covers real-world topics of 
societal concern in fields such as ecology, 
biology, social sciences, resource 
management and public health. When youth 
were asked how they would investigate a 
community problem, 80 percent suggested 
using computer modeling and simulation as a 
technique to investigate the issue. 

Career exploration is a primary focus of the 
girl-focused Techbridge program; all activities 
integrate a career exploration component. 
Techbridge extensively uses professional 
female role models who visit programs and 
share their experiences with participants; 
students also go on worksite field trips. Eighty 
percent of participants planned to take on 
additional STEM learning opportunities by 


electing advanced math and/or science 
classes. Eighty-five percent reported they find 
engineering more interesting and 83 percent 
said they find science more interesting. Eighty 
-one percent of participants said they can see 
themselves working in technology, science or 
engineering. 

While the afterschool field has made great strides 
toward offering strong afterschool STEM 
programming, much work remains to be done. 
Support systems are rapidly being assembled to 
support program providers, however assessing 
and documenting the impact of afterschool STEM 
experiences remains a challenge due to the 
diversity of programming. Efforts are being made 
to develop common frameworks for defining 
outcomes and assessments. Some common 
assessment tools have been developed to 
support the evaluation needs of many youth- 
serving organizations and to aggregate data 
across many programs. 

As we move forward, it is vital that local, state, 
and federal education policies are inclusive of the 
various contributors to the learning ecosystem; 
distributing the educational burden and 
opportunity across a range of appropriate 
organizations so it does not rest so heavily on 
schools. Policies must incentivize partnerships 
and encourage data sharing between partners. 
Resources must be allocated to reflect an 
understanding that the system as a whole is 
accountable for success and individual players 
are accountable for their role in contributing to 
this success. Fortunately, good progress has been 
made in the past few years toward these goals, 
though much remains to be done to pursue this 
essential agenda. 
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Introduction 


The last decade has seen growing interest in 
building the capacity of young people in science, 
technology, engineering and math (STEM) in the 
United States. The world has changed such that 
citizens require a greater level of STEM literacy to 
make decisions about complex issues such as 
global climate change, renewable energy sources 
and genetically modified foods. Increasingly, 
more and more jobs require proficiency in STEM 
and there is great concern that without access to 
adequate educational experiences, large 
segments of the population will be unable to 
participate effectively in the modern workplace. 
U.S. students are being outperformed in science 
and math by young people in many other 
industrialized countries 1 , which has alerted many 
of the need to improve this country's K-12 STEM 
education. 

Efforts to address the situation have primarily 
focused on education reform initiatives targeting 
schools. While schools and teachers are a critical 
part of young people's STEM education, 
increasingly they are seen as one part of a larger 
"learning ecosystem," one in which students are 
at the center of the system rather than a 
particular institution of learning taking center 
stage. In an ecosystem, the many components 
each have a differentiated and valuable role. The 
institutions that represent the components of a 
learning ecosystem work intentionally and 
collaboratively to create a rich educational world 


for children and youth as they grow into adults. 
The notion of a learning ecosystem, influenced by 
Bronfenbrenner's ecological theory of 
development (Bronfenbrenner, 1979), is not new. 
However, it is enjoying a renewed interest, 
particularly in the context of STEM learning (see 
the Research+Practice Collaboratory initiative 2 ; 
Barron, 2006; Hedegaard, 2009; Traphagen & 
Traill, 2014). 

Hence, as we consider redesigning learning to 
become more student-centered, we have to 
draw on our knowledge of how people learn and 
where people learn in order to design a system 
that more fully utilizes community assets, cultural 
institutions, and technological and media 
resources. The landmark 2009 study from the 
National Research Council, Learning Science in 
Informal Environments, highlighted this issue and 
examined informal learning settings and out-of- 
school-time (OST) programs. The report 
described an outcomes framework that included 
several strands of learning, two of which are 
particularly relevant for afterschool programs— 
one strand states that learners in informal 
environments will experience excitement, 
interest, and motivation to learn about 
phenomena in the natural and physical world; 
and the other states that learners will begin to 
think about themselves as science learners and 
develop identities as young people for whom 
science matters. 


1 29 countries outperformed the U.S. in mathematics and 22 countries outperformed the U.S. in science, as measured by 
the 2012 Program for International Student Assessment, or PISA. 

2 The Research+Practice Collaboratory initiative seeks to engage researchers and practitioners to address gap between 
educational research and practice in STEM education. The initiative takes an explicitly ecological perspective on learning, 
and hence has a primary focus on furthering research and practice that address learning across settings. For more 
information see www.researchandpractice.org 
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Some of the hesitation to change current models 
of teaching and learning may arise from a lack of 
knowledge, trust, and perhaps a lack of 
centralized control of the various spaces that 
children and youth inhabit. And yet, spaces such 
as afterschool and summer programs, libraries, 
science centers, and other such OST settings offer 
great resources and capabilities as partners in 
providing STEM learning experiences for youth. 
Many of these partners are sophisticated and 
experienced in designing programs that not only 
engage young people in STEM fields, but also 
help them to develop tangible skills. It is vital to 
acknowledge afterschool and other OST settings 
as necessary partners to the formal education 
system, and not just a luxury. 

The role of afterschool in STEM learning 

Afterschool programs (used here to broadly refer 
to before-school, afterschool and summer 
learning programs) are an important and growing 
part of the STEM education ecosystem in the 
United States. Roughly 8.4 million children 
participate in these programs each year, many 
from populations that are under-represented in 
STEM fields and careers. Children from ethnic 
backgrounds historically underrepresented in 
STEM attend in higher numbers— 24 percent of 
African-American, 21 percent of Hispanic and 16 
percent of Native American children attend 
afterschool programs, compared to the national 
average of 15 percent (Afterschool Alliance, 
2009). Data from the Department of Education's 
21 st Century Community Learning Centers 
program suggest that girls attend afterschool 
programs in equal numbers to boys. These 
programs serve an important role in providing a 
safe space for children while their parents are at 
work, but they are increasingly offering 
innovative learning opportunities. 



As schools have focused on measureable 
academic results in recent years, other aspects of 
schooling like music, art, learning healthy 
behaviors and engaging in physical activity are 
reduced or cut out of the school day due to 
budget pressures. These subjects, often 
considered 'enrichment' programming, are not 
seen as key aspects in preparing children for 
success. Afterschool programs have stepped up 
their efforts, anchored in a holistic youth 
development approach, to offer arts, sports and 
other learning programs that offer youth a voice, 
choice and control over their own learning. 

Youth-serving organizations such as 4-H, Girls 
Inc., and Girl Scouts have long embraced STEM in 
their program models. And in the past few years, 
the larger afterschool field has begun to 
recognize STEM as an important area of 
enrichment. Historically, afterschool educators 
have experimented with a learning approach that 
links school-day academic science and math with 
enriched learning in the afterschool hours. 

Often these connections were established 
through homework support, but increasingly 
STEM activities in the afterschool space are hands 
-on or project-based, serving to simultaneously 
engage young people in STEM topics, teach them 
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scientific reasoning and apply concepts learned 
during the school day. Today, many afterschool 
and summer programs are including STEM 
education as a standard part of their 
comprehensive programming. Thus, STEM has 
become an important element in a learning 
ecosystem rich in exploration, fun and hands-on 
activities that can lead to greater knowledge and 
interest in STEM coursework and careers. 

Does afterschool STEM have an impact? 

Practitioners closest to students typically answer 
this question affirmatively— they see youth 
engaged in and excited about STEM activities, 
asking significant questions, and wanting to learn 
more. However, funders and policy makers often 
seek data that substantiates such claims and 
demonstrates positive changes in a variety of 
outcomes— interest and engagement in science, 
greater knowledge of STEM careers, election of 
school science classes, and, sometimes, improved 
test scores in science and math. While some 
programs are able to achieve these outcomes 
and document their successes, it is unrealistic to 
expect that the afterschool field, in aggregate, 
will be able deliver on all of these outcomes 
immediately. Investments in afterschool STEM 
programs are still growing, as is the infrastructure 
that supports programs to offer more and 
higher-quality STEM opportunities. We can get a 
glimpse at the potential for what high-quality 
afterschool and summer STEM programs can 
offer by taking a look at what outcomes strong 
afterschool STEM programs are currently 
delivering. 

This paper begins by presenting some of the 
research findings about the importance of 
afterschool and other OST experiences for STEM 
learning. It then proceeds to summarize 


evaluation data from a selection of strong 
afterschool STEM programs and describes the 
types of substantive impacts these programs are 
having on participating youth. The paper 
concludes with recommendations for how we 
can construct an integrated approach to 
education that includes in-school and OST 
experiences. Appendices provide details of the 
programs whose outcomes are featured in the 
narrative. 
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Research on afterschool and out-of-school-time learning 


A well-established research base indicates the 
importance of cross-sector learning and 
underscores the importance of such experiences 
and activities outside of the classroom, especially 
for under-privileged populations (e.g., see Posner 
& Vandell, 1999; Bransford et al., 2006; Covay & 
Carbonaro, 2011). 

The capability of OST learning experiences, such 
as afterschool programs, to support the social 
and emotional growth of students, and the 
genesis of these programs as initiatives to 
positively influence students' personal 
development, is often overlooked in an 
educational culture that devotes significant 
attention to test scores and academic 
achievement. There are many studies that 
document the myriad ways afterschool programs 
impact youth. An oft-cited meta-analysis (Durlak 
& Weissberg, 2007) by the Collaborative for 
Academic, Social and Emotional Learning (CASEL), 
found that children participating in afterschool 
programs saw a significant improvement in their 
perceptions of themselves, improved positive 



social behavior and a decrease in problem 
behaviors. Vandell, Reisner and Pierce (2007) 
looked at 35 quality afterschool programs and 
found that students regularly participating in the 
programs improved their work habits; 
demonstrated higher levels of persistence; and 
saw reductions in reports of misconduct, such as 
skipping school. 

In addition to supporting a child's development 
and sense of worth, building social skills, and 
igniting his or her passion for learning, 
afterschool programs do have the ability to 
positively impact a child's academic performance. 
Both the CASEL meta-analysis and the Vandell et 
al. (2007) study found that, on average, students 
participating in quality afterschool programs 
show gains in their school-day performance. 
Several additional evaluation studies (see 
Afterschool Alliance, 2013b) also demonstrate 
the ability of afterschool programs to support the 
learning that takes place during the school day 
and help boost students' academic performance, 
and increase the likelihood of graduating from 
high school— especially for students who have 
fallen behind in school and need extra support 
and mentoring. 

New research by Deborah Vandell and colleagues 
(Auger, Pierce, & Vandell, 2013) finds that not 
only do students in afterschool programs see 
academic gains, but afterschool programs are 
also helping to close the achievement gap. The 
disparity in academic performance that has been 
documented between high- and low-income 
students disappears when low-income 
elementary school students have high levels of 
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participation in afterschool programs. The 
findings 3 also show that students participating in 
afterschool programs, especially those who 
regularly participate, see gains in their math 
achievement and other academic performance, 
improve their work habits, and have better 
school day attendance. 

More specifically, the math achievement gap 
between low- and high-income students 
narrows when low-income students attend 
afterschool programs with greater frequency. 

Conversely, the less often low-income students 
participate in afterschool programs, the larger 
the math achievement gap between them and 
their higher-income peers. This has significant 
implications for STEM education, as math 
proficiency is often the gatekeeper for many 
STEM fields. 

Research on the role of STEM in 
afterschool 

Lack of a STEM "identity" is often cited as one of 
the main reasons that young people do not 
pursue STEM fields, an issue highlighted in a 
recent study from King's College London 
(ASPIRES Project, 2014). The ASPIRES study 
found that most of the participating young 
people held relatively high career aspirations, 
but a very small percentage of the study sample 
(15 percent) aspired to be scientists. They 
reported liking science 4 , having positive views of 
science and scientists, and doing well in school 
in those subjects— but imagined scientists to be 
mostly white, middle-class, male and "brainy" 


and thus was not something they could imagine 
for their own lives. Most of these students and 
their families also did not understand that 
science could lead them to a diverse range of 
promising careers. This result is in keeping with 
the findings of an important paper by Tai et al. 
(2006), which showed that for students with 
average achievement in math, an early interest 
in pursuing physical science or engineering 
careers was a better predictor of whether they 
would follow through on that interest compared 
to their peers who had higher achievement in 
math but did not express a desire for such 
careers. 

Generating interest and engagement in STEM is 
a stated goal of many afterschool STEM 
programs. The focus is thus on nurturing 
curiosity and engagement with certain topics or 
fields (Krapp et al., 1992; Noam & Shah, 2013), 
and not necessarily on increased academic 
achievement. Of course, the theory of change 
implicit in much of the work is that increased 
interest and engagement will translate into 
improved achievement. Many afterschool 
providers do not see it as their direct goal to 
improve test score results, but instead they 
strive to increase involvement and exploration 
with STEM, decrease anxiety around STEM, and 
energize motivation. 

The approach taken by these program providers 
is supported by theory and research. Hidi & 
Renninger (2006) have described how curiosity 
and enjoyment are not only critical first steps, 
but integral to increased and continued 


3 The full report is forthcoming, however findings are summarized in "The achievement gap is real: New research shows 
afterschool is a real solution linked to closing the gap" retrieved from www.expandinRlearning.orR/docs/The% 
20Achievement%20Gap%20is%20Real.pdf. 

4 Historically, science interest and identity have been studied more extensively than the other fields in STEM. Although 
research is now catching up in engineering and technology education, science continues to be used as a proxy for STEM. 
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engagement with STEM activities and are part of 
a cycle where interest and proficiency feed and 
build on each other. The National Research 
Council's framework (discussed previously; 2009) 
begins with interest and engagement and 
progresses to a science identity. Development of 
a science or STEM identity involves multiple 
pieces: getting young people interested in STEM 
topics and professions; developing competence 
and a sense of confidence; and getting youth to 
envision themselves as contributors and 
participants in this enterprise. 

Such studies highlight the importance of 
providing young people with opportunities to 
engage in STEM fields and with professionals on 
their own terms and in ways that may not always 
be possible during the school day: to design 
open-ended projects that are not guaranteed to 
succeed, to work on hands-on projects that build 
STEM skills and hence confidence in their 
abilities, and to interact with STEM professionals 


so that young people can update their ideas 
about the types of people who work in these 
fields. 

A study by Dabney et al. (2011) supports this 
hypothesis: they conducted a retrospective 
examination of the association between 
university students' current interest in STEM 
careers and their participation in secondary OST 
STEM activities. The investigators found a strong 
positive correlation that showed participation in 
OST STEM activities was associated with STEM 
career interests, for both young men and young 
women, and by a similar amount. The ability of 
OST programs to engage young girls in STEM has 
been noted in other studies as well. For example, 
a study by Tan et al. (2013), which examined 
middle school girls' experiences in school-day 
science classes and OST science clubs, suggests 
that the positive science identity development 
that takes place within OST environments may 
impact girls' science trajectories and career goals. 
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In the course of leading an afterschool STEM 
program for middle school youth, Barton and Tan 
(2010) investigated the relationship between 
science learning, science identity, and student 
agency (which refers to the ability of individuals 
to engage in a field or practice as they choose). 
Their study underscores the need to provide 
opportunities for young people to exercise STEM 
agency in a socially appropriate context so they 
develop a sense of identity linked to the STEM 
enterprise. 

The importance of increased STEM "dosage" (or 
duration of exposure) was studied by Wai et al. 
(2010), who found that added exposure to 
opportunities for STEM engagement and 
learning, through individualized experiences such 
as competitions, led mathematically talented 
students to greater STEM achievement than their 
counterparts who lacked such opportunities. The 
results from the 2009 National Assessment of 
Educational Progress (NAEP) in science also 
showed that 4 th graders who participated in 
"hands-on science activities" and 8 th and 
12 th graders who did "science-related activities 
outside of school" showed a significant increase 
in test scores compared to those who did not. 
Further, it appears that the dosage mattered. 
Students involved almost daily in hands-on 
science, both in- and out-of-school, scored better 
on the 4 th grade assessment than those who only 
participated once or twice a month. Afterschool 
programs offer an important opportunity to 
increase the dosage of STEM experiences for 
young people. 



Noam et al. (2014) have taken this one step 
further and studied the impact of higher doses of 
STEM opportunities within afterschool 
programs 5 . Program dosage (hours per week and 
number of weeks) was consistently found, across 
multiple large datasets and analyses, to correlate 
with science interest scores. On average, children 
who spent more time involved in science 
activities reported significantly higher science 
scores than children with less participation. 
Additionally, self-reported interest in science and 
improved skills, like critical thinking, increased in 
youth program participants. 


5 For this study, Noam et al. (2014) used the assessment tool, the Common Instrument of Interest in Science, which is a 
survey used to assess student interest and engagement in science. For more information, see www.pearweb.org/tools/ 
commoninstrument.html. 
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What impacts do strong afterschool STEM programs 
have on youth? 


Afterschool programs that offer high-quality 
STEM learning experiences have a significant 
impact on participating youth— they cultivate 
students' interest in STEM and build STEM skills 
and proficiencies among diverse groups of young 
people. Specifically in relationship to interest, 
students in a recent study (Robertson & Noam, 
2014) reported significantly greater science 
interest in afterschool and summer programs 
rated high in quality compared to programs rated 
average or low in quality. Program quality in 
these two studies was measured by the 
Dimensions of Success (DoS) tool 6 , a program 
quality assessment rubric. This parallels findings 
about the effect of quality programming in 
relationship to other, non-STEM outcomes. 


However, there is no widely-adopted measure of 
program quality or of related outcomes used in 
afterschool and summer programs. So in order to 
examine the potential range of the impacts and 
outcomes afterschool STEM programs can have, 
we gathered outcomes data from a sample of 
programs that have a reputation for excellence. 
This is not intended as a representative sample of 
the afterschool field as a whole, but it does allow 
for an understanding of what strong programs 
can achieve. This is especially important because 
the OST STEM field is relatively young, and strong 
outcomes should only be expected from strong 
programs (Noam & Shah, in press). Table 1 
presents brief descriptions of the featured 
programs. More extensive descriptions can be 
found in Appendix 2. 


Table 1. Brief descriptions of the programs featured in this paper. 


Program 

Brief description 

4-H Tech Wizards 

Multiple states 

A 4-H afterschool program that provides targeted student groups with technology training. It is 
delivered in partnership with schools, volunteers, libraries, nonprofits and other community 
partners. 

Build IT 

U.S. and Canada 

An afterschool and summer curriculum for middle school youth offered by Girls Inc. to develop 
information technology (IT) fluency, interest in mathematics and knowledge of IT careers. 

Computer 

Clubhouse 

International 

An OST program in which youth from underserved communities work with adult mentors to 
explore their own ideas, develop new skills and build self-confidence through the use of 
technology. 

FUSE 

(Frontiers in Urban 
Science Exploration) 

A strategy used in seven cities to institutionalize engaging, inquiry-based informal STEM 
education nationally. FUSE is a project of Every Hour Counts and The After-School Corporation. 


6 Read more about the Dimensions of Success (DoS) tool here www.pearweb.org/tools/dos.html 
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Table 1, continued. Brief descriptions of the programs featured in this paper. 


Program 

Brief description 

Girlstart 

Austin, Texas 

An intensive afterschool and summer program for girls that provides free STEM programming 
every week throughout the school year at partner schools. 

Project GUTS 

Santa Fe, New Mexico 

An afterschool program in which middle school students learn cutting-edge computing 
methods to solve modern-day problems. 

Science Action Club 

San Francisco , California 

An afterschool program that offers hands-on science investigations and participation in citizen 
science projects all in a fun, club-like environment. 

Science Club 

Chicago , Illinois 

An afterschool program run in partnership between Northwestern University and the Boys & 
Girls Club of Chicago in which graduate student mentors lead small groups of students 
through designing and running hands-on science experiments. 

Science Minors Club 

Chicago , Illinois 

An outreach initiative of the Museum of Science and Industry aimed at increasing interest in 
science in underserved neighborhoods by engaging students in places where they already 
spend their time after school, such as community-based organizations and schools. 

Tech Reach 

Austin , Texas 

An outreach program of the Thinkery, a hybrid science and technology center/children's 
museum, focused on creative computing. 

Techbridge 

Bay Area, California 

An afterschool and summer program focusing on hands-on projects and career exploration to 
inspire girls in science, technology and engineering. 


Each program had its own methodology for 
collecting data on outcomes and impacts, 
however we have organized the findings of their 
evaluation results according to a framework of 
youth outcomes for STEM learning in afterschool 
that was developed in a recent consensus study 
(Afterschool Alliance, 2013a), representing the 
opinions of expert afterschool practitioners and 
other stakeholders. 

The consensus study concluded that afterschool 
STEM programs can help young people to: (a) 
develop an interest in STEM and STEM learning 
activities; (b) develop a capacity to productively 
engage in STEM learning activities; and (c) come 
to value the goals of STEM and STEM learning 
activities. The indicators of progress toward such 
outcomes include: 


1. Actively participating in STEM learning 
activities; 

2. Demonstrating curiosity about STEM topics, 
concepts or practices; 

3. Developing abilities to productively engage in 
STEM processes of investigation; 

4. Developing abilities to exercise STEM- 
relevant life and career skills; 

5. Coming to understand the role that STEM 
plays in solving societal issues; and 

6. Developing an increased awareness of STEM 
professions. 

Next, we summarize the main findings with 

respect to these six indicators. 
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Summary of impacts and outcomes 

Despite the specific nature of each of the 
afterschool programs and their evaluations, there 
were several themes that emerged that help us 
better understand the impacts of OST STEM in 
general. 

1. The programs are successful in engaging and 
retaining large numbers of students from 
diverse populations in STEM. 

All programs featured in this paper focus on 
engaging underserved youth in high-need 
communities. On average, 75 percent of students 
served are eligible for free or reduced price 
lunch 7 . High percentages of minority students are 
enrolled; an average of 82.5 percent of youth are 
of Latino, African-American, Native American or 
Asian/Pacific Islander 8 ethnic background 9 . Youth 
have many options for what they do after school, 
and young people consistently choose to attend 
these programs. They participate in high numbers 
and return regularly, reflecting a keen interest in 
the hands-on STEM learning they're being 
exposed to. 

For example: 

• Program attendance and engagement in the 
4-H Tech Wizards is high: on average 95 
percent of enrolled youth attend sessions 
regularly and 95 percent of participants 
stayed in the program for three years to 
complete all three skill levels. 


• Across more than 100 drop-in based 
Computer Clubhouse sites, 83 percent of 
participants visited their clubhouse at least 
weekly, and 47 percent every day. Ninety-one 
percent visited for at least one hour and 37 
percent of youth visited their clubhouse for 
more than three hours at a time. 

• In Project GUTS, middle school students are 
presented with a challenging computing 
curriculum and 82 percent persist in the 
program to complete a working computer 
simulation model (22 to 26 weeks). 

2. Young people in these programs express 
curiosity and interest in STEM subjects, in 

ways that extended that interest in school 
and out of school. 

For example: 

• Participation in Girlstart resulted in the 
extension of interest in science outside the 
program. Eighty-six percent of participants 
agreed with the statement "I want to try 
more science activities." Eighty-four percent 
reported interest in taking further STEM 
classes in middle or high school. In 2012, 58 
percent attended Girlstart's Girls in STEM 
conference. 

• Sixty-five percent of students in Project GUTS 
strongly agreed that participating in the 
program made them more excited to do and 
learn science and technology in school. 


7 7 of 11 programs reported data on the eligibility status for free or reduced price lunch of their participants. 

8 While Asian-Americans overall are not underrepresented in STEM, we chose to include Asian/Pacific Islander in this 
calculation because students in these programs came from such high-poverty backgrounds. The National Center for 
Science and Engineering Statistics at the National Science Foundation places Pacific Islanders in the "Other" category, 
which is underrepresented. 

9 8 of 11 programs reported data on the ethnic background of participants. 
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• Seventy-seven percent of youth in Science 
Action Club reported that "being in Science 
Action Club makes me want to learn more 
about science outside of school." 

3. As they participate in these programs, young 
people gain real skills and the ability to 
productively engage in STEM processes of 
investigation. 

Participants in these programs become familiar 
with concepts such as the engineering design 
process, scientific enquiry and problem solving. 
They apply these ideas to hands-on projects that 
include computer coding; Web development; 
designing hovercraft and solar cars; and building 
robots, geographic information systems (GIS) and 
other such technologies. 

For example: 

• Observations of Science Minors Club's staff 
revealed that 80 percent encouraged youth 
to formulate testable questions and 93 
percent fostered the collection of data and 
recording of observations. At 93 percent of 
sites, staff was observed providing 


opportunities for youth to use tools like a 
hand lens, calorimeter and rulers to make 
observations, take measurements or collect 
data. 

• Science Club's curriculum strongly 
emphasizes teaching scientific skills through 
the scientific method and engineering design 
process. Youth design their own experiments 
each week and collect data to make 
evidence-based conclusions. As a result, 
Science Club youth outperform their matched 
peers in Chicago Public Schools' annual 
science fair. Youth are assigned an initial 
aptitude category (low/middle/high) and the 
effect of Science Club participation is 
equivalent to raising youth scores by a full 
aptitude category (e.g. scores for Science 
Club youth in the middle aptitude category 
are statistically indistinguishable from non- 
participating high aptitude youth). 

• Youth in 4-H Tech Wizards are closely 
mentored by professionals specializing in 
emerging technologies and spend three years 
progressively developing their technology 
skills. As a result, 95 percent of 4-H Tech 
Wizards participants demonstrate mastery of 
skills in website development, video and 
podcast productions, GIS and GPS 
technologies, and LEGO robotics. 

• Techbridge girls consistently utilize the 
engineering design process. For example, 
they brainstorm on how to make a device to 
assist a disabled person, design a prototype 
and redesign the model based on use. 
Ninety-five percent of participants said they 
understand it can take many tries to solve a 
problem; 91 percent said they try harder to 
overcome a challenge; and 81 percent said 
they are better at problem-solving. More 
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than 93 percent said they know more about 
how things work, like circuits and simple 
machines, and 80 percent said they are 
better at using new computer programs. 

4. Youth learn essential STEM-relevant life and 
career skills, such as working in teams and 
collaborating effectively, as well as making 
presentations to audiences. 

Youth become demonstrably more confident 
about their abilities to work on complex tasks 
and projects as well as persistence in the face of 
initial failures. For example, 90 percent of 
Girlstart participants responded positively to the 
statement "I understand that it is okay if my 
Girlstart activity does not work on the first try." 
Youth in these programs also start to use 
scientific terminology appropriately, consider 
multiple perspectives, and even begin to 
understand the importance of peer-review in 
scientific research. 

Other examples: 

• All programs observed in FUSE received a top 
score (5/5) in an observation-based 
evaluation tool for providing opportunities to 
practice group process skills, such as actively 
listening, contributing ideas or actions to a 
group, doing a task with others, or taking 
responsibility for a part of the project. 

• Observations of Science Minors Club staff 
indicated that 100 percent utilized 
cooperative groups and individual roles to 
promote collaboration between youth 
participants. Eighty percent of observed sites 
provided opportunities for youth to report 
out their findings and communicate their 
ideas to the broader group and 86 percent 


supported youth in making connections 
between their work and their everyday lives. 

• Tech Reach students made statistically 
significant improvements in communication, 
collaboration and computing skills on an 
assessment of 21st century problem solving 
skills. Participants improved their confidence 
with technology— 90.6 percent of students 
reported that "I am good at computers" and 
more than 88 percent said "I like to figure 
out how something works." 

• Computer Clubhouse youth who visited 
more frequently and stayed longer exhibited 
higher levels of collaboration. Ninety-two 
percent of youth plan to use skills acquired 
in the clubhouse in their future careers. 
Eighty-two percent of alumni say they are 
currently using the tools and technologies 
they learned at the clubhouse professionally 
and/or personally. 

5. Equally importantly, participants in many of 
these programs come to understand the 
value of STEM in contributing to society and 
solving global and local problems. They begin 
to see how STEM intimately connects to 
their everyday lives. 

Other examples: 

• In observations of Science Minors Club sites, 
86 percent of facilitators supported youth in 
making connections to their everyday lives 
and 78 percent of participating youth 
indicated that they use science in their 
everyday lives. 

• Project GUTS curriculum covers real-world 
topics of societal concern in fields such as 
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ecology, biology, social sciences, resource 
management and public health. Participants 
demonstrate their understanding that STEM 
relates to everyday life and can be used to 
study and potentially solve local community 
problems through their individual projects 
and presentations. When asked how they 
would investigate a new community issue, 80 
percent suggested using computer modeling 
and simulation as a technique to investigate 
the issue. 

• Science Action Club youth are engaged with 
increasing scientific understanding of their 
local environments by contributing to 
authentic science in citizen science projects. 
Seventy-five percent of youth respondents 
agreed that "in Science Action Club, I do 
activities that contribute to real science 
research." Sixty-two percent agreed that 
"Science Action Club makes me wonder and 
ask questions about the natural world." 

6. Students in these programs display an 
increased awareness of career options, as 
well as a nuanced understanding of those 
careers. 

The Build IT curriculum, delivered through Girls, 
Inc., gives a powerful demonstration of the 
impact afterschool programs can have on young 
people's choice of careers. Girls in the program 
showed statistically significant improvement on 
the survey items related to knowing what classes 
to take in high school for an IT career. Interviews 
with female participants as well as afterschool 
staff document that participating in Build IT has 
made a noticeable difference in how girls view 
technology careers, including having a more 
positive image of technology careers, a more 



nuanced understanding of the types of careers 
that require knowledge and skills in technology 
and math, and— for many girls— a clearer picture 
of the coursework involved in training for an IT- 
influenced career. Results from the other 
featured programs echo these outcomes. Youth 
not only develop an interest in STEM topics, but 
also gain an understanding of what related 
careers might involve. 

Other examples: 

• Science Club participants are exposed to a 
wide array of STEM professions through 
curricular units covering topics ranging from 
neuroscience to food science. Graduate 
student mentors, who lead these curricular 
units, similarly come from diverse scientific 
disciplines. Youth participants often ask their 
mentors questions about their career path, 
life in the lab, and their lives outside of 
school. Participants regularly describe their 
favorite school subject, their dream jobs and 
their hobbies in program surveys. One 
hundred percent of students felt science was 
relevant to their future careers after 
participation in Science Club, up from 70 
percent before participation. Students could 
also more specifically describe STEM careers. 

• Each Girlstart After School lesson includes a 
discussion of a STEM career, including 
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educational prerequisites and how to enter 
that career. Ninety-four percent of 
participants demonstrated awareness that 
success in STEM can broaden their career 
options. Sixty-eight percent expressed strong 
interest and 93 percent indicated at least 
moderate interest in entering a STEM career. 
Almost all participants— 97 percent- 
expressed an intent to attend college after 
high school, which is notable among a 
population in which 55 percent are first 
generation college aspirants. 

A key component of Techbridge is its use of 
STEM professionals as role models, which 
provides opportunities for youth participants 
to engage in career exploration. As a result, 
94 percent knew more about different kinds 
of jobs. Eighty-one percent of girls said that 
because they engaged with role models and 
participated in field trips, they were more 
interested working in technology, science or 
engineering. 

High numbers of FUSE participants believe 
the program contributed to their science 
career aspirations— 79.5 percent reported 
that participating in their afterschool science 
program(s) "made the idea of a job in science 
when I am older seem more possible"; 73.2 
percent reported that it "made me more 
interested in a science job when I am older"; 
and 69 percent reported that it "made me 
feel more sure that I want a job in science 
when I am older." Eighty-eight percent of 
students reported that participating in FUSE 
"made me more confident that I could do 
well in science classes in college." 

Finally, afterschool STEM programs can have 

an impact on academic performance. 


A select number of programs have been able to 
track their students longitudinally. An evaluation 
of Project Exploration, a Chicago afterschool and 
summer STEM program, found that high-school 
graduation rates for participants are higher 
compared their peers, and their aspirations to 
attend college and major in a STEM field are 
higher (Chi et. al., 2010). An evaluation of 
Techbridge focused on long-term educational 
outcomes (2006-2012) found that participation 
helped girls achieve greater academic success. In 
comparisons made between girls who 
participated in Techbridge and those who did not 
(at the same school), findings indicated that: 

• Techbridge girls have a higher weighted total 
cumulative GPA at high school graduation 
(3.32) than girls who have not participated in 
Techbridge (2.94). 

• Techbridge girls scored an average of 26 
points higher (321 vs. 295) than non- 
Techbridge girls on the California Standards 
Algebra II test, and an average of 43 points 
higher (365 vs. 322) on the California 
Standards Biology test. 

• Girls who participated in Techbridge are 12 
percent more likely to enroll in Advanced 
Placement (AP) Calculus (AB or BC) than girls 
who have not. Furthermore, Techbridge 
participants have a higher average grade in 
calculus than non-participants (87 percent vs. 
82 percent). 

Recently, Girlstart released an analysis of the 
impact of their programs (Bussiere & Hudgins, 
2014). The primary questions for the analysis 
were: (1) Does Girlstart help girls perform better 
on standardized math and science tests? (2) Do 
Girlstart participants choose advanced STEM 
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classes and electives at higher rates than non- 
participating peers? 

• Girlstart After School participant test scores 
show that Girlstart girls are doing better than 
their non-participant peers in both math and 
science. 

0 In spring 2013, 76 percent of Austin 
Independent School District (AISD) 
Girlstart After School participants passed 
the 5 th grade State of Texas Assessments 
of Academic Readiness (STAAR) science 
test, while only 41 percent of comparison 
group members passed (64 percent of all 
students AISD partner schools passed- 
including both boys and girls). 

° Over a two-year period, 71 percent of 
AISD Girlstart participants passed the 5 th 
grade STAAR science test, while only 48 
percent of comparison group members 
(and 62 percent of all students) passed. 

0 On the STAAR math test, 85 percent of 
Girlstart participants passed, compared to 
70 percent of comparison group members 
(and 73 percent of all students at AISD 
partner schools). 

• This impact is not limited to AISD. In 2013, 87 
percent of Girlstart After School participants 
at the Georgetown Independent School 
District (GISD) partner schools passed 5 th 
grade math, compared to 77 percent of 
students overall at partner schools. Over a 
two-year period, 82 percent of Girlstart 
participants passed the 5 th grade STAAR test 
in science compared to 77 percent of 
students overall at GISD partner schools. 


• Girlstart participants are also more likely to 
enroll in advanced and pre-AP math and 
science— 1.58 advanced courses per girl, 
compared to 1.00 per non-participant girl. In 
conjunction with Techbridge's findings, this 
provides strong evidence that participation in 
OST STEM programs can be a significant 
factor in encouraging girls to pursue 
advanced math and science classes in school. 

Many other programs observe that student 
impacts extend to academics or the school day. 
However, such measures are difficult to capture 
due to the lack of resources available for 
longitudinal tracking of students and the 
difficulty in having school districts share student- 
level data. The most critical limitation in 
evaluation efforts of afterschool and other OST 
STEM programs is the availability of validated 
tools (Noam and Shah, 2013), as well as the 
amount of time and resources available for 
planning and implementing evaluation. 

As a final note, the featured programs had 
chosen, prior to the writing of this paper, which 
outcomes to assess and evaluate based on their 
defined programmatic goals and funder 
requirements. Additionally, they did not form 
their evaluation plans around the youth 
outcomes framework discussed above. The goal 
of this section was not to describe a 
representative sample of programs and students, 
but a nominated collection of strong programs 
and their impacts. Appendix 3 presents additional 
evaluation data from these programs set within 
the youth outcomes framework. 
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Conclusions 


Afterschool programs present a great 
opportunity to engage a diverse group of young 
people in the STEM fields. It may be tempting to 
view such programs as compensating for needs 
not currently met via formal education, but it is 
crucial to remember that these programs serve a 
unique role. Afterschool programs complement 
the learning that occurs during the school day 
with experiences rooted in youth development 
principles and have many impacts that go beyond 
STEM-specific outcomes. As discussed here, in- 
school and OST programs are one set of 
components in a learning ecosystem that 
includes family, media, and cultural institutions. 

Afterschool providers have embraced STEM in 
part because the hands-on learning it affords fits 
well with the core tenets of youth development, 
such as offering youth a voice and affording them 
opportunities to discover and grow their own 
interests. STEM programming in afterschool has 
grown at an extremely rapid pace in the past few 
years. Several support systems are now in place 
to aid both the growth in the number of 
programs offering STEM, as well as the quality of 
such programs. City-based intermediaries and 
statewide afterschool networks are able to 
coordinate resources and support programs; 
blended professional development models such 
as Click2Science PD 10 are helping to meet the 
professional development needs of afterschool 
educators; and investments in developing 
outcomes frameworks and assessment tools are 
greatly helping the field to set appropriate goals 


and document successes and challenges more 
systematically. 

The program evaluations and studies that are 
included in this paper point to important 
contributions afterschool STEM programs can 
make. However, the sample of programs 
described here is not representative of the 
afterschool field as a whole, but instead serves as 
a glimpse of the potential outcomes possible if 
the afterschool field is further engaged and 
strengthened. Leading youth-serving 

organizations such as 4-H, Big Brothers Big 
Sisters, Boys & Girls Clubs of America, Girls Inc., 
and the YMCA of the USA have recently joined 
forces to make STEM a priority. This initiative has 
the potential to bring quality curricula, activities 
and mentoring to millions of children and youth. 

It is difficult to isolate various environmental 
factors and attribute cause and effect specifically 
to afterschool interventions. Hence, there is a 
need to conduct further research to better 
understand these issues, specifically research 
that uses control groups in order to strengthen 
the evidence that positive changes can be 
attributed to the learning experience, and not 
simply a difference in interest among students 
who choose to participate in afterschool STEM. 
Absent such efforts, it will always be difficult to 
understand the selection bias of students who 
join programs and how it may differ from those 
who do not. In addition, these groups have to be 
studied within the same schools and districts in 


10 Click2Science is an online STEM professional development tool for out-of-school-time professionals and is available at 
www.click2sciencepd.org 
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order to distinguish effects that come from 
schools and those that are related to afterschool 
programs and other parts of the learning 
ecosystem. 

Nonetheless, the consistent findings across 
increasing numbers of afterschool programs give 
us confidence that we are seeing a real 
phenomenon: students participating in OST 
programs have immediate and long-term gains 
on a number of STEM-related dimensions. 

As the idea of cultivating learning ecosystems 
gains ground and afterschool programs are 
increasingly included as a key strategy in STEM 
education improvement, it is necessary to remain 
vigilant about the roles and values assigned to 
the components of an ecosystem. A true 
ecosystem assigns different roles to the different 
players, only some of which are duplicative. 
Local, state and federal policies and resource 
allocation must reflect an understanding that the 


system as a whole is accountable for success and 
all of the various players are accountable for their 
role in contributing to this success. The 
educational burden and opportunity cannot rest 
solely on schools; it must be distributed across a 
range of appropriate organizations. Policies must 
also incentivize partnerships between these 
organizations and encourage data to be shared 
among partners. Resources must be allocated for 
professional development to support program 
providers. Finally, there is a tremendous need to 
develop assessment instruments that can 
measure the impacts and outcomes of the 
system and the individual players appropriately. 
Progress has been made toward this goal, though 
much remains to be done to pursue this essential 
agenda. 
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Appendix I: Framework of Youth Outcomes 

The youth outcomes framework for afterschool STEM was developed in a consensus study by expert 
afterschool practitioners and stakeholders (Afterschool Alliance, 2013a). 


Outcome 

Indicators 

Sub-Indicators 

Through afterschool 
STEM programs, 
children and youth: 

You know or can see 
that children and 
youth demonstrate: 

If you had appropriate tools, you could document the following types of evidence: 

A. Develop an 
interest in 
STEM and 
STEM learning 
activities. 

"1 like to do this." 

Active 

participation in 
STEM learning 
opportunities 

Active engagement and focus in STEM learning activities 

Persisting in o task or program; sharing knowledge and ideas ; expressing enthusiasm , joy, etc. 

Pursuit of other out-of-school-time STEM learning opportunities 

Enrolling in programs; attending programs regularly; reporting performing STEM-related activities at home 

Pursuit of school STEM learning opportunities 

Participating more actively in school STEM activities; enrolling in courses; selecting special programs or 
schools; improving academic achievement 

Curiosity about 
STEM topics, 
concepts or 
practices 

Active inquiries into STEM topics, concepts or practices 

Exploring ideas verbally or physically; questioning, hypothesizing, testing 

Active information-seeking about mechanical or natural phenomena or objects 

Conducting internet searches for more information; getting books/journals about STEM; watching TV 
programs on science, etc. 

B. Develop a 
capacity to 
productively 
engage in 

Ability to 
productively 
engage in STEM 
processes of 
investigation 

Demonstration of STEM knowledge 

Demonstrating increase in knowledge in specific content areas; making connections with everyday world; 
using scientific terminology 

Demonstration of STEM skills 

Formulating questions; testing, exploring, predicting, observing, collecting and analyzing data 

Demonstration of an understanding of STEM methods of investigation 

Demonstrating understanding of the nature of science; using evidence-based reasoning and argumentation; 
demonstrating engineering design practices 

STEM learning 
activities. 

"1 can do this." 

Ability to 
exercise STEM- 
relevant life 
and career skills 

Demonstration of mastery of technologies and tools that can assist in STEM investigations 

Developing capacity to use measurement and other scientific instruments; running computer programs for 
data analysis; developing effective methods to communicate findings 

Demonstration of ability to work in teams to conduct STEM investigations 

Communicating effectively with team members; collaborating effectively with team members; 
demonstrating leadership on the team 

Demonstration of applied problem-solving abilities to conduct STEM investigations 

Engaging in critical thinking; questioning, sequencing, reasoning 

C. Come to value 
the goals of 
STEM and 
STEM learning 
activities. 

"This is important 
to me." 

Understanding 
of value of 
STEM in society 

Demonstration of an understanding of relevance of STEM to everyday life, including personal life 

Referencing examples of STEM in everyday life: everyday problems 

Demonstration of knowledge of important civic, global and local problems that can be addressed by STEM 

Contributing to projects that address a community need; developing awareness of how STEM is implicated in 
larger societal issues 

Demonstration of awareness of opportunities to contribute to society through STEM 

Engaging in a service-learning project 

Awareness of 
STEM 

professions 

Development of an understanding of the variety of STEM careers related to different fields of study 

Gaining knowledge about relevant professions; gaining knowledge of where such jobs and careers exist 

Demonstration of knowledge of how to pursue STEM careers 

Acquiring knowledge of what courses are needed to prepare for or pursue STEM degrees; declaring STEM 
interests or majors 

Demonstration of awareness that STEM is accessible to all 

Expressing a desire to meet role models; declaring STEM interests and majors; desiring to become a role 
model to pave the way for others 
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Appendix II: Descriptions of selected afterschool 

STEM programs with strong youth outcomes 



4-H Tech Wizards 

Build IT 

Computer Clubhouse 

Location 

Originally in Oregon; now in 26 states 

U.S. and Canada 

International (100 locations in 20 countries), 
headquartered at Museum of Science, Boston 

Description 

4-H Tech Wizards is an afterschool, small-group 
mentoring program that trains youth on various 
technologies within a bilingual, bicultural 
learning environment. The program is 
implemented through the 4-H National 
Mentoring Program, a partnership between 
National 4-H Council and the Office of Juvenile 
Justice and Delinquency Prevention. 

Build IT is an afterschool and summer 
curriculum for middle school youth to 
develop IT (information technology) 
fluency, interest in mathematics, and 
knowledge of IT careers. The program — 
co-developed by SRI International and 
the Girls Inc. of Alameda County— 
is designed to engage girls and African- 
American and Latino/a youth. 

The Computer Clubhouse provides a creative, safe 
out-of-school learning environment in which youth 
from underserved communities work with adult 
mentors to explore their own ideas, develop new 
skills, and build self-confidence through the use of 
technology. 

Age Level 

Ages 8-18; older youth involved as teen teachers 

Middle school 

Ages 10-18 

Students Served 

8,623 

3,500 

25,000 

Demographics 

Open to all, but focus is on Hispanic and other 
under-served, high-need youth. 

95% girls; more than 80% are African- 
American and Latina and the majority 
are from low-income households. 

Demographics depend on the local community. 
All clubhouses focus on reaching under-served 
communities and youth. 

Evaluation 

Methods 

Outcome data is collected by a variety of 
methods, including formal skill assessment tools; 
surveys; service logs; observations; and 
interviews with youth, mentors, staff and family 
members. 

Staff collects ongoing outcomes data 
through performance tasks, concept 
surveys and attitude surveys. Evaluation 
data is collected during specific research 
projects and includes participant, 
facilitator, staff and parent surveys, as 
well as interviews and observations with 
youth and facilitators. 

As a drop-in program, the clubhouse's first 
measure of success is participation. A sign-in 
system collects data about participants and 
attendance. An annual survey gathers 
demographic data; clubhouse visiting patterns; 
and attitudes related to technological competence, 
academic engagement, social-emotional well- 
being, and aspirations for the future. 
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FUSE 


Girlstart 


Project GUTS 

Santa Fe Institute 


Location 

New York, New York; Providence, Rhode Island; Oakland, 
California; Baltimore, Maryland; Boston, Massachusetts; 
Chicago, Illinois; and Palm Beach County, Florida 

Austin, Texas, and other high-need 
communities 

Santa Fe, New Mexico 

Description 

FUSE is a strategy to institutionalize engaging, inquiry- 
based, informal STEM education nationally. The goal is to 
stimulate a culture shift among afterschool leaders and 
staff to increase the demand for and capacity to deliver 
high-quality informal STEM education. 

Girlstart After School is an intensive 
intervention program providing free weekly 
STEM programming throughout the year at 
partner schools. Girlstart programs are 
designed to promote girls' early engagement 
and academic success in STEM, encourage 
their aspirations and persistence in STEM 
education and careers, and incubate a 
talented and diverse STEM workforce. 

Project GUTS (Growing Up Thinking 
Scientifically) is an afterschool program 
in which middle school students learn 
cutting-edge computing methods to 
solve modern-day problems. 
Participants design, create and test 
computer models to simulate "what if" 
scenarios for real-world questions of 
community and societal concern. 

Age Level 

Grades K-12 

Ages 8-13; Grades 4-8 

Ages 11-13; Grades 6-8 

Students Served 

32,000 

1,000 

2,000 

Demographics 

Varies by city 

100% girls; 70% qualify for federal free or 
reduced price lunch; 33% are Limited English 
Proficient; 5% have special needs or 
disabilities; 12% African-American, 5% Asian- 
American/Pacific Islander, 17% Caucasian, 
63% Latina, 4% identify as multiracial; 55% 
first-generation college aspirants. 

35% qualify for federal free or reduced 
price lunch; 7% are Limited English 
Proficient; 5% have special needs or 
disabilities; 1% African-American, 2% 
Asian-American, 24% Caucasian, 58% 
Hispanic/Latino, 2% Native American, 
13% Other. 

Evaluation 

Methods 

The FUSE initiative is evaluated by TASC (The After School 
Corporation), which conducts interviews with stakeholders; 
collects surveys from staff, students and intermediaries' 
partners; and observes science activities using the STEM 
Program Quality Assessment (PQA). Staff confidence is 
examined via the Science Teaching Efficacy Belief 
Instrument (STEBI), and youth science attitudes via the 
Science Attitude Change Tool and Common Instrument. 

Girlstart uses a system of pre- and post- 
surveys to assess girls' STEM skills and 
knowledge; self-reported attitudes toward 
Girlstart; and interest and confidence in 
undertaking future STEM activities, courses, 
majors and careers. A framework for 
tracking program alumnae's academic 
progress assesses long-term impact. 

Project GUTS measures student 
demographics and attendance; 
collects pre- and post-surveys of 
knowledge, skills and self-efficacy; 
teacher participation in professional 
development; growth and self-efficacy 
of teachers as GUTS club leaders; and 
partnerships developed. 
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Science Action Club 

California Academy of Sciences 


Science Club 


Science Minors Clubs 

Museum of Science and Industry 


Location 

San Francisco, California 

Chicago, Illinois 

Chicago, Illinois 

Description 

In Science Action Club (SAC), youth conduct 
dynamic and authentic investigations by 
exploring their local environment, making 
observations about the natural world, and 
contributing data to real scientific research. 
SAC also provides resources for other 
afterschool partners to implement the 
program. 

Science Club is a partnership between 
Northwestern University and the Boys & 
Girls Club of Chicago, utilizing long-term 
mentoring relationships to engage low- 
income urban youth in science. Every week, 
Northwestern graduate students lead small 
groups of students through designing and 
running hands-on science experiments. 

Science Minors Clubs is an outreach initiative of 
the Museum of Science and Industry aimed at 
increasing interest in science in underserved 
neighborhoods by engaging students in places 
where they already spend their time after school, 
such as community-based organizations and 
schools. Participants work together on STEM 
projects and activities that build curiosity and 
encourage teamwork. 

Age Level 

Ages 11-14; Grades 6-8 

Grades 6-8 

Ages 8-12 

Students Served 

430 (expansion planned for fall 2014) 

60 

6,370 

Demographics 

66% qualify for federal free or reduced price 
lunch; 25% are Limited English Proficient; 
18% have special needs or disabilities; 15% 
African-American, 50% Asian-American/ 
Pacific Islander, 13% Caucasian, 

13% Hispanic/Latino, 9% Other. 

97% qualify for federal free or reduced price 
lunch; 35% are Limited English Proficient; 
15% have special needs or disabilities; 

40% Asian-American/Pacific Islander, 

33% African-American, 5% Caucasian, 

22% Hispanic/Latino. 

87% qualify for federal free or reduced price lunch; 
48% African-American, 6% Asian-American/Pacific 
Islander, 8% Caucasian, 36% Hispanic/Latino. 

Evaluation 

Methods 

SAC has partnered with a professional 
evaluation firm to assess programmatic 
impact on participating youth and activity 
leaders. Outcome data is collected through 
pre- and post-session surveys, site visits, 
interviews and focus groups. In addition to 
developing original assessment strategies, 
the program employs validated evaluation 
tools. 

Using a case-control methodology, Science 
Club measures changes in youth skills with 
a scenario-based skills interview and 
science fair scores. Youth attendance, 
interviews and surveys are also collected. 
Mentors participate in focus groups and 
self-report on changes in communication 
and teaching skills as well as attitudes 
toward science outreach. 

The museum's in-house research and evaluation 
team leads the design of the evaluation plan. 

Data sources include student intake forms, weekly 
attendance reporting, surveys, facilitator reflections 
and site observations. Program implementation 
across diverse sites, effectiveness and usability of 
the curriculum and pedagogical approaches, impact 
of the professional development trainings, and 
community perceptions of the museum have also 
been evaluated. 
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Tech Reach 

The Thinkery 


Techbridge 


Location 

Austin, Texas 

Bay Area, California; expanding to Seattle in fall 2014 and two additional 
cities by 2016 

Description 

Tech Reach is an outreach program of the Thinkery, a hybrid 
science and technology center/children's museum formerly known 
as the Austin Children's Museum. Tech Reach engages students 
through a concept called "creative computing"— where the bridge 
to computing concepts is achieved through projects and challenges 
that tap into young students' creativity and inventiveness. 

Techbridge offers afterschool and summer programs with hands-on 
projects and career exploration to inspire girls in science, technology and 
engineering. Curricula are developed with girls in mind, and designed to 
spark and sustain an interest in these fields as well as to make a connection 
with STEM careers. Career exploration is facilitated through STEM role 
model visits and field trips to local engineering and technology companies. 

Age Level 

Grades 3-5 

Grades 5-12 

Students Served 

150 

400 

Demographics 

95% qualify for federal free or reduced price lunch; 10% are 
Limited English Proficient; 15% have special needs or disabilities; 
10% African-American, 2% Asian-American/Pacific Islander, 

5% Caucasian, 80% Hispanic/Latino. 

100% girls; 94% of school partners receive Title 1 funding; 75% of students 
are eligible for free or reduced price lunch; 40% Limited English Proficient; 
10% African-American, 30% Asian-American/Pacific Islander, 8% Caucasian, 
42% Hispanic/Latino, 2% Native American, 8% Other. 

Evaluation 

Methods 

An external evaluator designs pre- and post-program assessments, 
embedded assessments, and surveys for students to measure 
attitudes toward STEM and the program. Focus groups are 
convened for participating teachers to help determine program 
strengths in design and implementation and identify areas for 
improvement. 

Techbridge uses quantitative and qualitative evaluation methods that 
include pre- and post-surveys; focus groups with girls, teachers, and 
families; and program observations and coaching. Surveys examine 
technical and scientific ability, career awareness, aspirations, teamwork 
and adult influence. Comparison groups are utilized to evaluate outcomes 
with groups not participating in Techbridge. In addition, an observation 
rubric is used to ensure fidelity of implementation and identify areas for 
program improvement. 
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Appendix III: Selected evaluation results of 

afterschool STEM programs with 
strong youth outcomes 


Each afterschool program utilized the youth outcomes framework (Afterschool Alliance, 2013a; also 
shown in Appendix I) to describe their youth-level impacts. Programs collected a variety of evaluation 
data, however only data that fit the framework is shown here. Additionally, many programs found 
anecdotal evidence supporting the outcomes and indicators, but did not collect data on all of them. 
This evidence is not reflected in the following tables. 


Outcome A: Develop an interest in STEM and STEM learning activities 

"1 like to do this." 

Indicators: 

Active participation in STEM learning opportunities 

Curiosity about STEM topics, concepts or practices 

4-H Tech 
Wizards 

Program attendance is high and consistent— on 
average, 95% of enrolled youth attend 4-H Tech 
Wizard sessions and 95% of participants have stayed 
in the program for three years to complete all three 
skill levels. 

Youth report that they are attracted to 4-H Tech 
Wizards because of the opportunities to work with 
cutting edge technology and the mentors. 

Build IT 

Retention was often 100% at sites that had girls and 
parents commit to girls' active participation in Build 
IT. Girls reported a statistically significant increase in 
their confidence in math, belief in its usefulness, 
and plans to take computer courses. 


Computer 

Clubhouse 

83% of participants visited their clubhouses at least 
weekly, and 47% every day. 37% of youth visited 
their clubhouse for more than three hours at a time, 
and 91% visited for at least one hour. 

Clubhouse alumni reported that they initially 
started attending to learn about technology (61%) 
and work on interesting projects (45%). Continued 
participation gave them the opportunity to work 
on projects of their own (38%), learn important 
skills (34%), and gave them access to valuable 
technology (31%). 

FUSE 

FUSE students participated in additional science- 
related opportunities— 43.6% played a math or 
science game at home; 42% participated in 
discussions about science topics with friends; 
55% watched TV, movies or online videos related 
to science topics; and 30% read a book about a 
science topic. 

Student attitudes toward science increased 
significantly in terms of agreement with the 
following statements: "I get excited to find out 
that 1 will be doing a science activity"; "Science is 
something 1 get excited about"; "1 like to work on 
science activities"; "1 like to participate in science 
projects"; and "1 am curious to learn more about 
science, computers, or technology." 
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Outcome A: Develop an interest in STEM and STEM learning activities 

"1 like to do this." 

Indicators: 

Active participation in STEM learning opportunities 

Curiosity about STEM topics, concepts or practices 

Girlstart 

Participation resulted in the extension of interest in 
science outside the program— 86% agree with the 
statement "1 want to try more science activities/' 
84% reported interest in taking further STEM 
classes in middle or high school. In 2012, 58% 
attended Girlstart's Girls in STEM Conference. 

87% report enjoying science, and particularly the 
scientific activities they did at Girlstart. 

Project GUTS 

82% of Project GUTS youth persist in the program to 
successfully complete a working computer model. 
65% of students strongly believed that participating 
in Project GUTS made them more excited to do and 
learn science and technology in school. 


Science 
Action Club 

69% of participants attend the program regularly 
and 86% indicated that they would recommend it to 
a friend. More than 60% of youth who enrolled in 
two sessions also participated in a third session. 

77% of youth reported that "Being in Science Action 
Club makes me want to learn more about science 
outside of school." 

74% agree that "1 get excited to find out that 1 will 
be doing a science activity in afterschool." 63% 
agree that "science is one of my favorite subjects in 
afterschool." 

Science Club 

Youth choose to go to Science Club among an array 
of options at a Boys & Girls Club— 84% of 
participants attend weekly, participating for 1.5 
years on average, with a 92% retention rate. 

94% of youth prefer learning hands-on science with 
Science Club over school-based science classes. 

82% of youth express a desire to continue in the 
program after as a high school mentor. 

Science 
Minors Club 

87% of youth surveyed indicated that they enjoy 
science and 92% expressed interest in doing more 
science activities. Family engagement is strong— on 
average, 1266 participant family members attend 
Museum of Science and Industry Family Days. 

85% of site facilitators reported an interest in 
additional STEM topics to meet youth participant 
demand. 57% of site facilitators accessed additional 
curriculum kits to expand offerings for their club 
participants. 

Tech Reach 


After participating in Tech Reach, 88% of students 
believed that science was fun and 86% said that 
math was fun. 

Techbridge 

After participation, 80% of girls planned to pursue 
additional STEM learning opportunities by taking 
advanced math and/or science classes. 

85% of girls reported they find engineering more 
interesting and 83% said they find science more 
interesting after participating in Techbridge. 
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Outcome B: Develop a capacity to productively engage in STEM learning activities 

"1 can do this." 

Indicators: 

Ability to productively engage in STEM 
processes of investigation 

Ability to exercise STEM-relevant life and career skills 

4-H Tech 
Wizards 

95% of 4-H Tech Wizards participants 
demonstrate mastery of skills in website 
development, video and podcast productions, 
GIS and GPS technologies, and LEGO robotics. 


Build IT 

The computer engineering design process is 
embedded in every curriculum unit as a 
method of problem solving. Girls showed 
statistically significant improvements in their 
understanding and use of the design process. 

Build IT motivates girls to use technology to strengthen 
and build their technology fluency. The project 
achieved statistically significant improvements in 
frequency of computer use, computer skills, and 
conceptual understanding of computing. 

Computer 

Clubhouse 

Youth who visited more frequently showed 
higher levels of problem-solving competence 
and technology competence. Of students in 
that group, 61% of girls and 55% of boys 
scored above the median on "technology 
competence." 

Clubhouse youth who visit more frequently and stay 
longer show higher levels of collaboration. 92% of 
youth plan to use skills acquired in the clubhouse in 
their future careers. 82% of alumni say they are 
currently using the tools and technologies they learned 
at the clubhouse professionally and/or personally. 

FUSE 

Youth saw gains in science knowledge, 
motivation and confidence across all FUSE 
sites participating in the evaluation. At least 
85% of youth reported that participating in 
their afterschool science program: "Improved 
my understanding of science"; "Helped me 
learn things that 1 need to answer science 
questions"; and "Gave me experience that 
will help me in the future with science 
projects and activities." 

FUSE received a top score (5/5) on an observation- 
based evaluation for providing opportunities to practice 
group process skills, which includes actively listening, 
contributing ideas or actions to a group, doing a task 
with others, or taking responsibility for a part of a 
project. 

Girlstart 

100% engage weekly in iterative design and 
scientific inquiry to solve problems, including 
generating observations, predictions and 
hypotheses, and tracking experiments and 
outcomes. As a result, 91% of participants 
demonstrated mastery of scientific inquiry 
and the engineering design process. 

100% participate in weekly collaborative problem- 
solving challenges that require critical thinking. 90% 
respond positively to the statement "1 understand that 
it is okay if my Girlstart activity does not work on the 
first try." All participants present their experiments and 
results to parents, teachers and others in an end-of- 
semester showcase. 

Project GUTS 

64% of youth agreed they had learned how to 
use computer models to conduct scientific 
investigations— a skill that requires following 
an iterative problem solving process to 
design, implement, test and debug computer 
models. 
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Outcome B: Develop a capacity to productively engage in STEM learning activities 

"1 can do this." 

Indicators: 

Ability to productively engage in STEM 
processes of investigation 

Ability to exercise STEM-relevant life and career skills 

Science 
Action Club 

100% of participants contribute directly to a 
national citizen science project by making 
observations, collecting data and considering 
the implications of their findings. 72% of 
youth report that "The data 1 collect in 
Science Action Club is scientifically relevant." 

The SAC environment emphasizes teamwork and 
leadership, and evaluations show gains in youth self- 
perception and leadership. As a result of an emphasis 
on using technology for data collection, 89% of 
participants agreed that, "1 feel confident using 
technology to do science activities in SAC." 

Science Club 

The Science Club curriculum emphasizes the 
scientific method and the engineering design 
process. Science Club youth significantly 
outperform their aptitude-matched peers in 
two independent, well-controlled oral 
assessments of science skills. These 
assessments include constructs of 
experimental design, use of variables, and 
data analysis. Science Club youth are also 
more confident conducting experiments than 
non-participating youth (56% vs. 32%). 

Twice as many Science Club students described 
experiments as a way to help them learn and find out 
new things compared to youth not in the program 
(64% vs. 33%). 81% of Science Club youth described 
using science outside of school compared to the 
control group. 

Science 
Minors Club 

Site observations revealed that 80% of 
facilitators encouraged youth to formulate 
testable questions and 93% fostered the 
collection of data and recording of 
observations. 93% of observed sites provided 
opportunities for youth to use STEM tools 
such as a hand lens, calorimeter and rulers 
to make observations, take measurements 
or collect data. 

Observations indicated that 100% of program sites 
utilized cooperative groups and individual roles to 
promote collaboration between youth participants. 
80% of sites observed provided opportunities for youth 
to report out their findings and communicate their 
ideas to the broader group. 

Tech Reach 

All students must use critical thinking and 
creative problem solving needed to design, 
program and build technology in Tech 
Reach— 77% of students report "1 am good at 
building things using technology." 

Students made statistically significant improvements 
in communication, collaboration and computing skills 
on an assessment of 21st century problem solving 
skills. 90.6% of students reported that "1 am good at 
computers" and more than 88% percent said "1 like to 
figure out how something works. 

Techbridge 

95% of girls say they understand it can take 
many tries to solve a problem. 93.5% said 
they know more about how things work, like 
circuits and simple machines. 80% said they 
are better at using new computer programs. 

70% said they are more comfortable speaking in 
front of a group of people. 91% said they try harder 
to overcome a challenge. 81% said they are better at 
problem-solving. 92% said they feel more confident 
using technology. 89% said they feel more confident 
in science. 
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Outcome C: Come to value the goals of STEM and STEM learning activities 


"This is important to me." 

Indicators: 

Understanding the value of STEM in society 

Awareness of STEM professions 

4-H Tech 
Wizards 

85% of 4-H Tech Wizards participants completed 
15 hours of community service learning by 
teaching technology to others. 

Evaluators report that 70% of graduating 4-H Tech 
Wizards participants pursued post-high school 
education and careers in science, technology, 
engineering or math. 

Build IT 


Interviews document that participation has made a 
noticeable difference in how girls view technology 
careers. Many who initially reported IT as solitary 
and boring later reported that they found IT to be 
collaborative, fun, intellectually stimulating and a 
possible career. Girls showed statistically significant 
improvements in knowing what classes to take in 
high school to prepare for an IT career. 

Computer 

Clubhouse 

74% of alumni give back to their communities 
through continued interaction with clubhouse 
staff and youth. 

Among alumni, 80% reported that the clubhouse 
had been the most important source of support for 
pursuing a career. 97% of alumni said that it was 
also the most important source of support for 
setting high goals and expectations for themselves. 

FUSE 

After participating in FUSE, student attitudes 
increased significantly in terms of agreement with 
the statement: "1 pay attention when people talk 
about recycling to protect our environment." 

79.5% reported that participating in FUSE "made 
the idea of a job in science when 1 am older seem 
more possible"; 73% reported that it "made me 
more interested in a science job when 1 am older"; 
and 69% reported that it "made me feel more sure 
that 1 want a job in science when 1 am older." 88.6% 
reported that participating in FUSE "made me more 
confident that 1 could do well in science classes in 
college." 

Girlstart 

After participating in Girlstart, 70% report that 
they believe their ideas can solve problems in 
their own neighborhoods. 87% realize that they 
use science frequently outside of Girlstart. 91% 
agree with the statement, "If 1 do well in STEM in 
college, 1 am more likely to get a better job." 

94% of participants demonstrate awareness that 
success in STEM can broaden their career options. 
68% express strong interest and 93% indicate at 
least moderate interest in entering a STEM career. 
In a population in which 55% of participants are 
first generation college aspirants, 97% expressed 
intent to attend college after high school. 
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Outcome C: Come to value the goals of STEM and STEM learning activities 

"This is important to me." 

Indicators: 

Understanding the value of STEM in society 

Awareness of STEM professions 

Project GUTS 

When asked how they would investigate a new 
community issue, 80% of Project GUTS 
participants suggested using computer modeling 
and simulation. 


Science 
Action Club 

In SAC, youth are engaged with increasing 
scientific understanding of their local 
environments. 75% of youth agreed that 
"In [SAC], 1 do activities that contribute to 
real science research." 62% agreed that "[SAC] 
makes me wonder and ask questions about the 
natural world." 68% agree that "1 care about 
the topics we learn about in [SAC]." 

SAC is central to the California Academy of 
Science's pathway of youth programs, which 
fosters an awareness of STEM professions. 40% of 
survey respondents agreed that "Being in SAC 
makes me think about getting a science job when 
1 am older." 

Science Club 

Science Club youth are better able to describe 
how science is used in everyday life in comparison 
to their non-participating peers. 

100% of students felt science was relevant to their 
careers after participation in Science Club, up from 
70% at the start of the program. Students could 
also more specifically describe science careers. 

Science 
Minors Club 

In site observations, 86% of facilitators supported 
youth in making connections to their everyday 
lives and 78% of participating youth indicated that 
they use science in their everyday lives. Of the 
sites implementing the "Green Energy" 
curriculum, 100% of facilitators reported 
participation in a recycling program. 

An average of 1,350 youth and family members 
participated in Museum of Science and Industry 
STEM career fairs. 100% of youth attending the 
fairs completed a STEM career journal, which 
guided their interviews of STEM professionals and 
encouraged them to describe STEM careers of 
personal interest. 

Tech Reach 


Thinkery staff members promote awareness of 
STEM careers and futures by routinely encourag- 
ing students to imagine themselves as engineers 
solving problems. 76% of participants reported 
"Someday 1 would like a job where 1 can use math 
and science." 

Techbridge 


Techbridge provides extensive opportunities to 
engage in career exploration. As a result, 94% of 
participants knew more about different kinds of 
jobs. 81% said they can see themselves working in 
technology, science or engineering. 
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